Detect Spark driver completion by container state when tracking via k8s API - #68048

I still need to test this out and write unit tests.

This has passed all unit tests as well as a manual test that I ran.

The original issue described a second problem

index 0 is not guaranteed to be the driver container in a multi-container pod.

This is reachable whenever driver_container is None and I think the PR does not address this? It’s fine to be fixed separately, but in that case the PR can’t close the issue outright.

The original issue described a second problem

index 0 is not guaranteed to be the driver container in a multi-container pod.

This is reachable whenever driver_container is None and I think the PR does not address this? It’s fine to be fixed separately, but in that case the PR can’t close the issue outright.

I had initially decided to leave that part of the code as is because it only acts as a fallback when the driver container cannot be identified. However, now I am thinking it might be more accurate to just report the pod failure without the exact container status even in case of the fallback. Something like this:

if phase == "Succeeded":
    terminal_phase = phase
    break
if phase == "Failed" and not container_completed:
    raise RuntimeError(f"Spark application {app_id} failed (phase=Failed)")

Or we could print the statuses of all containers in the pod without assuming any specific one is the driver. @uranusjr what do you think?

I can create a separate PR for this. I have edited this PR description so that it dos not say that the issue is closed.

I think it makes sense to just print the containers’ status. Probably safer and more helpful. This isn’t only a fallback though, it might also be reached when the driver is identified but hasn’t terminated yet. A pod can report phase=Failed while the driver container is still running since a sidecar died first. So maybe

• When driver_container resolved, report that container status by name.
• When it didn’t, list all statuses without claiming which is the driver.